fix: add missing commas in LLM prompt JSON formats and guard list return types - #274

Summary

Two related robustness issues that cause crashes when LLMs return malformed or unexpected JSON responses.

Changes

Fix #257 — Missing commas in LLM prompt reply formats

Six prompt templates in pageindex/page_index.py were missing a trailing comma after the "thinking" field:

• check_title_appearance (line 34)
• check_title_appearance_in_start (line 62)
• toc_detector_single_page (line 112)
• check_if_toc_extraction_is_complete (line 132)
• check_if_toc_transformation_is_complete (line 150)
• detect_page_index (line 213)

When an LLM follows the format literally, it produces invalid JSON (missing comma between keys). extract_json then fails and returns {}, causing KeyError on toc_detected, completed, or page_index_given_in_toc.

Fix #199 — generate_toc_init / generate_toc_continue must return a list

When extract_json fails on a malformed LLM response it returns {} (a dict). Both generate_toc_init and generate_toc_continue passed this through directly. Callers expect a list:

• process_no_toc calls toc_with_page_number.extend(...) → AttributeError: 'dict' object has no attribute 'extend'
• meta_processor calls item.get('physical_index') on list items → AttributeError: 'str' object has no attribute 'get'

Both functions now return [] when extract_json returns a non-list, matching the expected contract.

Testing

Added tests/test_llm_response_robustness.py with 5 tests covering:

1. All "thinking" fields in prompt reply formats have trailing commas
2. generate_toc_init returns [] (not a dict) when LLM returns malformed JSON
3. generate_toc_continue returns [] (not a dict) when LLM returns malformed JSON
4. generate_toc_init passes through a valid list unchanged
5. process_no_toc does not raise AttributeError when generate_toc_init returns []

Run with:

pytest tests/test_llm_response_robustness.py -v

Related Issues

Fixes #257
Fixes #199